1. Field of the Invention
The present invention relates to information processing apparatuses that integrate a plurality of feature parameters, and in particular, to an information processing apparatus in which, when speech recognition based on speech and on an image of lips observed when the speech was made is performed, the information processing apparatus increases speech recognition performance by integrating audio and image feature parameters so that the parameters can be processed in optimal form.
2. Description of the Related Art
By way of example, speech is recognized by extracting feature parameters from the speech, and comparing the feature parameters with normal parameters (normal patterns) used as a reference.
When speech recognition based on only speech is performed, there is a certain limit to increasing the recognition factor. Accordingly, it is possible that the speech recognition be performed based not only on the speech but also on a captured image of lips of the speaker.
In this case, it is also possible to integrate feature parameters extracted from the speech and feature parameters extracted from the lip image to form so-called xe2x80x9cintegrated parametersxe2x80x9d and to use the integrated parameters to perform speech recognition. The assignee of the present patent application has proposed Japanese Patent Application No. 10-288038 (which was not open to the public when the present patent application was filed) as a type of speech recognition that generates integrated parameters by integrating feature parameters extracted from speech and feature parameters extracted from a lip image and that uses the integrated parameters to perform speech recognition.
With reference to FIGS. 1 to 16, Japanese Patent Application No. 10-288038 is described below.
FIG. 1 shows an example of a speech recognition apparatus that performs speech recognition based on integrated parameters obtained by integrating feature parameters based on a plurality of input data.
In addition to speed data (as a speech from a user) to be recognized, image data obtained by capturing an image of the user""s lips when the user spoke, noise data on noise in an environment where the user spoke, and data useful in recognizing the user""s speech (speech), such as a signal in accordance with the operation of an input unit for inputting a place where the user speaks in the case where the speech recognition apparatus is provided with the input unit, are sequentially input in time series to the speech recognition apparatus. The speech recognition apparatus takes these types of data into consideration, as required, when performing speech recognition.
Specifically, the speech data, the lip-image data, the noise data, and other data, which are in digital form, are input to a parameter unit 1. The parameter unit 1 includes signal processors 111 to 11N (where N represents the number of data signals input to the parameter unit 1). The speech data, the lip-image data, the noise data, and other data are processed by the signal processors 111 to 11N corresponding thereto, whereby extraction of feature parameters representing each type of data, etc., is performed. The feature parameters extracted by the parameter unit 1 are supplied to an integrated parameter generating unit 2.
In the parameter unit 1 shown in FIG. 1, the signal processor (lip-signal processor) 111 processes the lip-image data, the signal processors (audio-signal processors) 112 to 11Nxe2x88x921 process the speech data, and the signal processor (audio-signal processor) 11N processes the noise data, etc. The feature parameters of the speech (sound) data such as the speech data and the noise data include, for example, linear prediction coefficients, cepstrum coefficients, power, line spectrum pairs, and zero cross. The feature parameters of the lip-image data include, for example, parameters (e.g., the longer diameter and shorter diameter of an ellipse) defining an ellipse approximating the shape of the lips.
The integrated parameter generating unit 2 includes an intermedia normalizer 21 and an integrated parameter generator 22, and generates integrated parameters by integrating the feature parameters of the signals from the parameter unit 1.
In other words, the intermedia normalizer 21 normalizes the feature parameters of the signals from the parameter unit 1 so that they can processed having the same weight, and outputs the normalized parameters to the integrated parameter generator 22. The integrated parameter generator 22 integrates (combines) the normalized feature parameters of the signals from the intermedia normalizer 21, thereby generating integrated parameters, and outputs the integrated parameters to the matching unit 3.
The matching unit 3 compares the integrated feature parameters and normal patterns (a model to be recognized), and outputs the matching results to a determining unit 4. In other words, the matching unit 3 includes a distance-transition matching unit 31 and a spatial distribution matching unit 32. The distance-transition matching unit 31 uses a distance-transition model (described below) to perform the matching of the integrated feature parameters by using a distance-transition method (described below), and outputs the matching results to the determining unit 4. The spatial distribution matching unit 32 performs the matching of the integrated feature parameters by using a spatial distribution method (described below), and outputs the matching results to the determining unit 4.
The determining unit 4 recognizes the user""s speech (sound), based on outputs from the matching unit 3, i.e., the matching results from the distance-transition matching unit 31 and the spatial distribution matching unit 32, and outputs the result of recognition, e.g., a word. Accordingly, in the determining unit 4, what is processed by speech recognition is a word. In addition, for example, a phoneme, etc., can be processed by speech recognition.
With reference to the flowchart shown in FIG. 2, processing by the speech recognition apparatus (shown in FIG. 1) is described below.
When the speech data, the lip-image data, the noise data, etc., are input to the speech recognition apparatus, they are supplied to the parameter unit 1.
In step S1, the parameter unit 1 extracts feature parameters from the supplied data, and outputs them to the integrated parameter generating unit 2.
In step S2, the intermedia normalizer 21 (in the integrated parameter generating unit 2) normalizes the feature parameters from the parameter unit 1, and outputs the normalized feature parameters to the integrated parameter generator 22.
In step S3, the integrated parameter generator 22 generates integrated feature parameters by integrating the normalized feature parameters from the intermedia normalizer 21. The integrated feature parameters are supplied to the distance-transition matching unit 31 and the spatial distribution matching unit 32 in the matching unit 3.
In step S4, the distance-transition matching unit 31 performs the matching of the integrated feature parameters by using the distance-transition method, and the spatial distribution matching unit 32 performs the matching of the integrated feature parameters by using the spatial distribution method. Both matching results are supplied to the determining unit 4.
In step S5, based on the matching results from the matching unit 3, the determining unit 4 recognizes the speech data (the user""s speech). After outputting the result of (speech) recognition, the determining unit 4 terminates its process.
As described above, the intermedia normalizer 21 (shown in FIG. 1) normalizes the feature parameters of the signals from the parameter unit 1 so that they can be processed having the same weight. The normalization is performed by multiplying each feature parameter by a normalization coefficient. This normalization coefficient is found by performing learning (normalization-coefficient learning process). FIG. 3 shows an example of a learning apparatus for performing the learning.
For brevity of description, a type of learning is described below that finds normalization coefficients for setting the feature parameters of the speech and the image as two different media (e.g., feature parameters of speech and feature parameters of lips observed when the speech was made) to have the same weight.
In FIG. 3, image feature parameter Pi,j and speech feature parameter Vi,j, which are code-vector learning parameters (codebook-creating data) for creating a codebook for use in vector quantization, are supplied to a tentative normalizer 51. The tentative normalizer 51 tentatively normalizes image feature parameter Pi,j and speech feature parameter Vi,j by using normalization coefficients from a normalization coefficient controller 55, and supplies the normalized feature parameters to a codebook creator 52. In other words, in order to use the weight of image feature parameter Pi,j as a reference and to set the weight of speech feature parameter Vi,j to equal the reference, speech feature parameter Vi,j is multiplied by normalization coefficient a from the normalization coefficient controller 55. Accordingly, it can be considered that image feature parameter Pi,j is multiplied by 1 as normalization coefficient xcex1.
In FIG. 3, suffix xe2x80x9cixe2x80x9d indicating the row of feature parameter Pi,j or Vi,j represents a time (frame) at which the feature parameter Pi,j or Vi,j was extracted, and suffix xe2x80x9cjxe2x80x9d indicating the column of feature parameter Pi,j or Vi,j represents the order (dimensions) of the feature parameter Pi,j or Vi,j. Therefore, (Pi,j, Pi,2, . . . , Pi,L, Vi,1, Vi,2, . . . , Vi,M) are feature parameters (feature vectors) at time i. Expression P(k)i,j formed by adding a suffix in parentheses to image feature parameter Pi,j represents a feature parameter generated from different learning data if xe2x80x9ckxe2x80x9d differs. This also applies to the suffix (k) of expression V(k)i.
The codebook creator 52 creates a codebook for use in vector quantization by a vector quantizer 54, using code-vector learning parameters Pi,j and Vi,j, and supplies it to the vector quantizer 54.
In the codebook creator 52, the codebook is created in accordance with, e.g., the LBG (Linde, Buzo, Gray) algorithm. However, another type of algorithm other than the LBG algorithm may be employed.
The LBG algorithm is so-called xe2x80x9cbatch learning algorithmxe2x80x9d, and locally converges code vectors (representative vectors) constituting the codebook in optimal positions by repeatedly performing Voronois division that optimally divides a feature parameter space in accordance with the distance between a feature parameter as a learning sample (learning data) and each code vector (a proper initial value is first given), and repeatedly updating the code vectors to the centroids of partial regions of a feature parameter space which are obtained by the Voronois division.
Here, when a set of learning samples is represented by xj (j=0, 1, . . . , Jxe2x88x921), and a set of code vectors is represented by Y={y0, y1, . . . , yNxe2x88x921}, learning-sample set xj is divided into N subsets Si (i=0, 1, . . . , Nxe2x88x921) by code-vector set Y in the Voronois division. In other words, when the distance between learning-sample set xj and code vector yi is represented by d (xj, yi), and the following expression holds with respect to all of t (t=0, 1, . . . , Nxe2x88x921) that does not equal i,
d(xj,yi) less than d(xj,yt)xe2x80x83xe2x80x83(1)
it is determined that learning-sample xj is (xj, Si) belonging to subset xj.
In addition, when centroids C (v0, v1, . . . , vMxe2x88x921) with respect to vectors v0, v1, . . . , vMxe2x88x921 are defined by the following expression:                               C          ⁡                      (                                          v                0                            ,                              v                1                            ,              …              ⁢                              xe2x80x83                            ,                              v                                  M                  -                  1                                                      )                          =                                            arg              ⁢                              xe2x80x83                            ⁢              min                        v                    ⁢                      {                                          1                M                            ⁢                                                ∑                                      m                    =                    0                                                        M                    -                    1                                                  ⁢                                  xe2x80x83                                ⁢                                  d                  ⁡                                      (                                          v                      ,                                              v                        m                                                              )                                                                        }                                              (        2        )            
code vector yi is updated in accordance with the following expression
yi=C({Si})xe2x80x83xe2x80x83(3)
In the expression (2), the right side xe2x80x9cargmin { }xe2x80x9d means vector v that minimizes the value in { }. A so-called xe2x80x9cclustering techniquexe2x80x9d using expression (3) is called xe2x80x9ck-means clusteringxe2x80x9d. The details of the LGB algorithm are described in, for example, xe2x80x9cSpeech and Image Engineeringxe2x80x9d written by Kazuo Nakata and Satoshi Minami, published by Shokodo in 1987, pp. 29-31.
In the learning apparatus shown in FIG. 3, suffix i,j that indicates the row of element Si,j and Ti,j in the codebook output by the codebook creator 52 represents the j-th element of the code vector corresponding to code #i. Thus, expression (Si,1, Si,2, . . . , Si,L, Ti,1, Ti,2, . . . , Ti,M) represents a code vector corresponding to code #i. Element Si,j of the code vector corresponds to the image, and element Ti,j corresponds to the speech.
A tentative normalizer 53 is supplied with image feature parameter Pi,j and speech feature parameter Vi,j (in this example it is assumed that both types of parameters are obtained from an image and speech different from those for the code-vector learning parameters) as normalization-coefficient learning parameters for learning normalization coefficient xcex1. Similarly to the tentative normalizer 51, the tentative normalizer 53 tentatively normalizes image feature parameter Pi,j and speech feature parameter Vi,j by using the normalization coefficients from the normalization coefficient controller 55, and supplies the normalized parameters to the vector quantizer 54. In other words, among image feature parameter Pi,j and speech feature parameter Vi,j as normalization-coefficient learning parameters, speech feature parameter Vi,j is multiplied by normalization coefficient a from the normalization coefficient controller 55 by the tentative normalizer 53, and the tentative normalizer 53 outputs the product to the vector quantizer 54.
The tentative normalizer 53 is supplied with a plurality of sets of normalization-coefficient learning parameters. The tentative normalizer 53 performs normalization with respect to each of the normalization-coefficient learning parameters.
The vector quantizer 54 performs vector quantization on the normalized normalization-coefficient learning parameters supplied from the tentative normalizer 53, using the latest codebook supplied from the codebook creator 52, and supplies quantization errors caused by the vector quantization to the normalization coefficient controller 55.
In other words, the vector quantizer 54 calculates, for the image and speech, a distance between each code vector of the codebook and each normalized normalization-coefficient learning parameter, and supplies the calculated shortest distance as a quantization error to the normalization coefficient controller 55. Specifically, the distance between image feature parameter Pi,j among the normalized normalization-coefficient learning parameters, and image-related element Si,j of the code vector, is calculated, and the calculated shortest distance is supplied as an image-related quantization error to the normalization coefficient controller 55. At the same time, the distance between speech feature parameter xcex1Vi,j among the normalized normalization-coefficient learning parameters, and speech-related element Ti,j of the code vector, is calculated, and the calculated shortest distance is supplied as a speech-related quantization error to the normalization coefficient controller 55.
The normalization coefficient controller 55 accumulates, with respect to all the normalization-coefficient learning parameters, image- and speech-related quantization errors supplied from the vector quantizer 54, and changes normalization coefficient xcex1 to be supplied to the vector quantizers 51 and 53 so that both accumulated values are equal.
With respect to the flowchart shown in FIG. 4, a normalization-coefficient learning process performed by the learning apparatus shown in FIG. 3 is described below.
In the learning apparatus shown in FIG. 3, at first, code-vector learning parameters are supplied to the vector quantizer 51, and normalization-coefficient learning parameters are supplied to the vector quantizer 53. In addition, initial normalization coefficient a is supplied from the normalization coefficient controller 55 to the vector quantizers 51 and 53.
In step S21, the vector quantizer 51 tentatively normalizes the code-vector learning parameters by multiplying speech feature parameter Vi,j among the code-vector learning parameters by normalization coefficient a from the normalization coefficient controller 55, and supplies the tentatively normalized parameters to the codebook creator 52.
When receiving the normalized code-vector learning parameters from the vector quantizer 51, the codebook creator 52 uses the received parameters in step S22 to create, based on the LBG algorithm, a codebook used when the vector quantizer 54 performs vector quantization. The codebook creator 52 supplies the created codebook to the vector quantizer 54.
In step S23, the tentative normalizer 53 tentatively normalizes the normalization-coefficient learning parameters by multiplying speech feature parameter Vi,j among the normalization-coefficient learning parameters by normalization coefficient xcex1 from the normalization coefficient controller 55, and supplies the tentatively normalized parameters to the vector quantizer 54.
When receiving the latest codebook from the codebook creator 52, and receiving the latest normalized normalization-coefficient learning parameters from the tentative normalizer 53, the vector quantizer 54 uses the codebook from the codebook creator 52 in step S24 to perform vector quantization for the image and the speech. The vector quantizer 54 supplies the image- and speech-related quantization errors to the normalization coefficient controller 55.
In other words, in step S24, the vector quantizer 54 calculates a distance between image feature parameter Pi,j (among the normalized normalization-coefficient learning parameters) and image-related element Si,j of the code vector, and supplies the calculated shortest distance as an image-related quantization error to the normalization coefficient controller 55. The vector quantizer 54 also calculates a distance between speech feature parameter xcex1Vi,j (among the normalized normalization-coefficient learning parameters) and speech-related element Ti,j of the code vector, and supplies the calculated shortest distance as a speech-related quantization error to the normalization coefficient controller 55.
As described, the vector quantizer 53 is supplied with the normalization-coefficient learning parameters. Thus, the vector quantizer 54 is also supplied with a plurality of sets of normalized normalization-coefficient learning parameters. The vector quantizer 54 successively finds, for each of the normalized normalization-coefficient learning parameters, the above-described image- and speech-related quantization errors, and supplies them to the normalization coefficient controller 55.
In step S24, the normalization coefficient controller 55 accumulates, for all the normalization-coefficient learning parameters, the image- and speech-related quantization errors supplied from the vector quantizer 54, thereby finding image-related quantization-error-accumulated value DP and speech-related quantization-error-accumulated value DV. The obtained image-related quantization-error-value DP and speech-related quantization-error-accumulated value DV are supplied and stored in the normalization coefficient controller 55.
In step S25, the normalization coefficient controller 55 determines whether image-related quantization-error-accumulated value DP and speech-related quantization-error-accumulated value DV have been obtained with respect to all the values of normalization coefficient xcex1. In other words, in this example, accumulated values DP and DV are found by, for example, initially setting normalization coefficient a at 0.001, and changing (increasing (in this example)) normalization coefficient xcex1 by 0.001 between 0.001 and 2.000. In step S25, the normalization coefficient controller 55 determines, for the image and the speech, whether quantization-error-accumulated values DP and DV have been found with respect to normalization coefficient xcex1 having the range.
If the normalization coefficient controller 55 has determined in step S25 that quantization-error-accumulated values DP and DV have not been found with all the values of normalization coefficient xcex1, the normalization coefficient controller 55 changes normalization coefficient xcex1 in step S26, as described above, and supplies it to the tentative normalizers 51 and 53. After that, the normalization coefficient controller 55 proceeds back to step S21, and uses the changed values of normalization coefficient xcex1 to repeatedly perform the same processing.
If the normalization coefficient controller 55 has determined in step S25 that quantization-error-accumulated values DP and DV have been found with all the values of normalization coefficient xcex1, it proceeds to step S27, and calculates the absolute value |DPxe2x88x92DV| of the difference between image-related quantization error DP and speech-related quantization error DV (stored in step S24) with respect to each value of normalization coefficient xcex1. The normalization coefficient controller 55 also detects the value of normalization coefficient xcex1 that gives the minimum value of difference absolute value |DPxe2x88x92DV|. In other words, the normalization coefficient controller 55 ideally detects normalization coefficient xcex1 in the case where image-related quantization error DP and speech-related quantization error DV are identical. The normalization coefficient controller 55 proceeds to step S28, and terminates the process after outputting normalization coefficient xcex1 giving the minimum value of absolute value |DPxe2x88x92DV|, the output normalization coefficient xcex1 set for performing normalization so that image feature parameter Pi,j and speech feature parameter Vi,j can be treated having the same weight.
As described above, a codebook is created by normalizing code-vector learning parameters as integrated parameters composed of image and speech feature parameters, and using the normalized code-vector learning parameters, while performing the steps of tentatively normalizing normalization-coefficient learning parameters as integrated parameters composed of image and speech feature parameters, finding accumulated values of image- and speech-related quantization errors (minimum values of distances with the code vectors) by using the created codebook to perform vector quantization on each of image and speech feature parameters among the normalized normalization-coefficient learning parameters, and changing normalization coefficients so that image- and speech-related accumulated values are equal. Thereby, normalization coefficients for performing normalization so that feature parameters of different media such as image and speech can be treated having the same weight can be found.
As a result, when speech recognition is performed by using normalization coefficients to normalize feature parameters extracted from speech and feature parameters extracted from an image of lips of the speaker, integrating the feature parameters, and using the integrated parameters, the recognition is greatly affected by either the speech or the image. This can prevent an increase in the recognition factor from being hindered.
In addition, effects of the feature parameters (of the media) which constitute the integrated parameters, on the recognition factor, can be easily verified.
In the above-described case, the weights of the image feature parameters are used as a reference (set to be 1), and normalization coefficient xcex1 for setting the weights of the speech feature parameters to be identical to those of the image feature parameters is found. Therefore, the intermedia normalizer 21 (shown in FIG. 1) outputs the image feature parameters without performing any processing, while it normalizes the speech feature parameters by multiplying the speech feature parameters by the normalization coefficient xcex1 found as described above, and outputs the normalized speech feature parameters.
Although the learning that finds normalization coefficient xcex1 for setting the weights of the feature parameters of two types (image and speech) to be equal has been described with reference to FIG. 3, a type of learning can be performed that finds normalization coefficients for equalizing the weights of feature parameters of three or more types or the weights of feature parameters of media other than the image and the speech.
The above-described normalization coefficient learning can be applied regardless of the type and order of feature parameters because it is not dependent on the type and order of feature parameters.
FIG. 5 shows an example of the distance-transition matching unit 31 shown in FIG. 1.
From the integrated parameter generating unit 2 (shown in FIG. 1), for example, integrated parameters generated when a word was pronounced are in time series supplied to a time-domain normalizer 61. The time-domain normalizer 61 performs time-domain normalization on the supplied, integrated parameters.
When a speech time in which a word was pronounced is represented by t, a time change of an element among integrated parameters generated when the word was pronounced is as shown in, for example, FIG. 6A. Speech time t in FIG. 6A varies depending on each speech, even if the same person pronounced the same word. Accordingly, the time-domain normalizer 61 performs time-domain normalization so that speech time t is uniformly set to be time TC, as shown in FIG. 6B. Assuming that the speech recognition apparatus (shown in FIG. 1) performs word recognition, time TC is set to be sufficiently longer than a general speech time required when a word to be recognized is pronounced. Thus, the time-domain normalizer 61 changes the integrated parameter shown in FIG. 6A so that it is so-called xe2x80x9cextendedxe2x80x9d in the time-domain direction. The technique of the time-domain normalization is not limited to that shown in FIGS. 6A and 6B.
The time-domain-normalized parameters are supplied from the time-domain normalizer 61 to a vector quantizer 62. The vector quantizer 62 sequentially performs vector quantization on the time-domain-normalized integrated parameters, using a codebook stored in a codebook storage unit 63, and sequentially supplies a distance calculator 64 with codes as the vector quantizer results, that is, codes corresponding to code vectors nearest to the integrated parameters.
The codebook storage unit 63 stores the codebook, which is used when the vector quantizer 62 performs vector quantization on the integrated parameters.
The distance calculator 62 accumulates, in units of time, each distance between a distance-transition model of the word to be recognized and a code vector obtained when a code series output by the vector quantizer 62 is observed, and supplies the accumulated value to a sorter 66.
A distance-transition-model storage unit 65 stores a distance-transition model representing distances between time-series integrated parameters (normal series) of the word to be recognized, which are as shown in FIG. 7, and the code vectors of the codebook stored in the codebook storage unit 63. In other words, the distance-transition-model storage unit 65 stores a distance-transition model (as shown in FIG. 7) that is obtained by learning (described below) for each word to be recognized.
In the example shown in FIG. 7, the codebook stored in the codebook storage unit 63 has J+1 code vectors C0 to CJ.
The sorter 66 selects upper Nb values (where Nb represents a natural number) in increasing order among distance-accumulated values on the distance-transition model of each word to be recognized, and outputs them, as a result of matching between the integrated parameters and the distance-transition model, to the determining unit 4.
The above-described, distance-transition matching unit 31 performs matching based on a distance-transition method. A matching process based on this distance-transition method is described below with reference to the flowchart shown in FIG. 8.
When receiving time-series integrated parameters corresponding to the pronunciation of a word from the integrated parameter generating unit 2 (shown in FIG. 1), the time-domain normalizer 61 performs time-domain normalization on the integrated parameters in step S31, and outputs the time-domain-normalized parameters to the vector quantizer 62. In step S32, the vector quantizer 62 sequentially performs vector quantization on the time-domain-normalized parameters supplied from the time-domain normalizer 61 by referring to the codebook stored in the codebook storage unit 63, and sequentially outputs a code series corresponding to code vectors having the shortest distances with the integrated parameters, as the vector-quantization results to the distance calculator 64.
In step S33, the distance calculator 64 accumulates each distance between the distance-transition model of the word to be recognized and each code vector obtained when the code series output by the vector quantizer 62 is observed.
In other words, when among the code series output by the vector quantizer 62, a code at time t is represented by St (t=0, 1, . . . , TC), the distance calculator 64 finds the distance between the code and code vector Cj (j=0, 1, . . . , J) corresponding to code s0 initially output by the vector quantizer 62 by referring to the distance-transition model. Specifically, when code s0 corresponds to, for example, code vector C0, the distance at time #0, which is on the curve indicating the distance transition from code vector C0, is found in FIG. 7.
The distance calculator 64 calculates the distance at time #1 to code vector Cj corresponding to code s1 secondly output by the vector quantizer 62 by referring to the distance-transition model. Similarly, the distance calculator 64 sequentially finds distances up to the distance at time #TC to code vector Cj corresponding to code STC finally output by the vector quantizer 62 by referring to the distance-transition model, and calculates an accumulated value of the distances.
After calculating accumulated values of distances for all distance-transition models stored in the distance-transition-model storage unit 62, the distance calculator 64 outputs the accumulated values to the sorter 66, and proceeds to step S34.
In step S34, the sorter 66 selects upper Nb values in increasing order among the accumulated values of distances on the distance-transition models of words to be recognized, and proceeds to step S35. In step S35, the sorter 66 outputs, to the determining unit 4, the selected values as a result of matching between the integrated parameters and the distance-transition models.
With reference to FIG. 9, a learning apparatus for performing learning that finds the distance-transition models to be stored in the distance-transition-model storage unit 62 (shown in FIG. 5) is described below.
A time-domain normalizer 71 is supplied with time-series, learning integrated parameters. The time-domain normalizer 71 performs time-domain normalization on the learning integrated parameters, similarly to the time-domain normalizer 61 (shown in FIG. 5), and supplies the normalized parameters to a distance calculator 72.
In other words, the time-domain normalizer 71 is supplied with, for example, a plurality of sets of time-series, learning integrated parameters for finding a distance-transition model of a word. The time-domain normalizer 71 performs time-domain normalization on each of the learning integrated parameters, and processes the normalized parameters to generate one learning integrated parameters. Specifically, a plurality of learning integrated parameters (Nc learning integrated parameters in FIG. 10) (on a word) that do not always have the same duration are supplied to the time-domain normalizer 71, as shown in column (A), FIG. 10. The time-domain normalizer 71 performs time-domain normalization on the supplied parameters so that each of their durations is set to be time TC, as shown in column (B), FIG. 10. The time-domain normalizer 71 calculates, for example, the mean of values sampled at the same time from the time-domain-normalized parameters, as shown by the graph (C) in FIG. 10, and generates one learning integrated parameter in which the calculated mean is used as a value sampled at each point of time.
Processing of a plurality of learning integrated parameters for generating one learning integrated parameter is not limited to the method described with reference to FIG. 10. If only a (time-series) learning integrated parameter for finding a distance-transition model of a word is provided, the learning integrated parameter is simply time-domain-normalized and output by the time-domain normalizer 71.
The distance calculator 72 sequentially calculates the distance between each code vector stored in the codebook storage unit 73 and each of the learning integrated parameters supplied from the time-domain normalizer 71, and supplies a polynomial unit 74 with each distance transition, i.e., transitions of distances (as shown in FIG. 11) between learning integrated parameters at time #0 to time #TC and the code vectors.
The codebook storage unit 73 stores a codebook identical to the codebook stored in the codebook storage unit 63 shown in FIG. 5.
The polynomial unit 74 finds a polynomial expression approximating the distance transitions supplied from the distance calculator 72, and outputs it as a distance-transition model. In other words, the polynomial unit 74 performs approximation of the distance transitions (shown in FIG. 11), using the curves shown in FIG. 7, which are expressed in a polynomial expression.
With reference to the flowchart shown in FIG. 12, processing by the learning apparatus shown in FIG. 9 is described below.
A plurality of (time-series) learning integrated parameters for finding a distance-transition model on a word are supplied to the time-domain normalizer 71.
In step S41, the time-domain normalizer 71 performs time-domain normalization on each of the supplied parameters, and processes the normalized parameters to generate one learning integrated parameter. The learning integrated parameter is output to the distance calculator 72.
In step S42, the distance calculator 72 sequentially calculates each distance between each code vector stored in the codebook storage unit 73 and the learning integrated parameter supplied from the time-domain normalizer 71, and supplies the polynomial unit 74 with transitions of the calculated distances.
In step S43, the polynomial unit 74 finds an N-th order polynomial expression for approximation of the distances between the learning integrated parameters (normal series) and the code vector, using, for example, least squares.
In other words, Nd-th order polynomial expression fj(t) for approximation of the distance between a learning integrated parameter at time t and code vector Cj is expressed by the following expression:
fj(t)=aj0+aj1t1+aj2t2+ . . . +ajNdtNdxe2x80x83xe2x80x83(4)
Thus, by solving the following expression, using distance fj(t) (obtained by the distance calculator 72) between the learning integrated parameter at time t and code vector Cj, coefficients aj0, aj1, aj2, . . . , ajN defining expression (4) can be found using least squares.                                                                         (                                                                                                    a                                                  j                          ⁢                                                      xe2x80x83                                                    ⁢                          0                                                                                                                                                                        a                                                  j                          ⁢                                                      xe2x80x83                                                    ⁢                          1                                                                                                                                                ⋮                                                                                                                          a                                                                              j                            ⁢                                                          xe2x80x83                                                        ⁢                            N                            ⁢                                                          xe2x80x83                                                        ⁢                            d                                                    -                          1                                                                                                                                                                        a                                                  j                          ⁢                                                      xe2x80x83                                                    ⁢                          N                          ⁢                                                      xe2x80x83                                                    ⁢                          d                                                                                                                    )                            =                              xe2x80x83                            ⁢                                                (                                                                                                              ∑                                                      xe2x80x83                                                    ⁢                                                      t                            Nd                                                                                                                                                ∑                                                      xe2x80x83                                                    ⁢                                                      t                                                          Nd                              -                              1                                                                                                                                                  ⋯                                                                                              ∑                                                      xe2x80x83                                                    ⁢                                                      t                            1                                                                                                                                                ∑                                                      xe2x80x83                                                    ⁢                                                      t                            0                                                                                                                                                                                        ∑                                                      xe2x80x83                                                    ⁢                                                      t                                                          Nd                              +                              1                                                                                                                                                                            ∑                                                      xe2x80x83                                                    ⁢                                                      t                            Nd                                                                                                                      ⋯                                                                                              ∑                                                      xe2x80x83                                                    ⁢                                                      t                            2                                                                                                                                                ∑                                                      xe2x80x83                                                    ⁢                                                      t                            1                                                                                                                                                                                                                                                  ⋮                                                                                                                                                                          ∑                                                                  xe2x80x83                                                                ⁢                                                                  t                                                                                                            2                                      ⁢                                                                              xe2x80x83                                                                            ⁢                                      Nd                                                                        -                                    1                                                                                                                                                                                                                                                                                                                                      ⋮                                                                                                                                                                          ∑                                                                  xe2x80x83                                                                ⁢                                                                  t                                                                                                            2                                      ⁢                                      Nd                                                                        -                                    2                                                                                                                                                                                                                                                                                                                                      ⋰                                                                                                                                                                          xe2x80x83                                                                                                                                                                                                                                                                    ⋮                                                                                                                                                                          ∑                                                                  xe2x80x83                                                                ⁢                                                                  t                                  Nd                                                                                                                                                                                                                                                                                                    ⋮                                                                                                                                                                          ∑                                                                  xe2x80x83                                                                ⁢                                                                  t                                                                      Nd                                    -                                    1                                                                                                                                                                                                                                                                                                                    ∑                                                      xe2x80x83                                                    ⁢                                                      t                                                          2                              ⁢                                                              xe2x80x83                                                            ⁢                              Nd                                                                                                                                                                            ∑                                                      xe2x80x83                                                    ⁢                                                      t                                                                                          2                                ⁢                                Nd                                                            -                              1                                                                                                                                                  ⋯                                                                                              ∑                                                      xe2x80x83                                                    ⁢                                                      t                                                          Nd                              +                              1                                                                                                                                                                            ∑                                                      xe2x80x83                                                    ⁢                                                      t                            Nd                                                                                                                                )                                                  -                  1                                                                                                                        xe2x80x83                            ⁢                              (                                  xe2x80x83                                ⁢                                                                                                    ∑                                                  xe2x80x83                                                ⁢                                                                                                            f                              j                                                        ⁡                                                          (                              t                              )                                                                                ⁢                                                      t                            0                                                                                                                                                                                                  ∑                                                  xe2x80x83                                                ⁢                                                                                                            f                              j                                                        ⁡                                                          (                              t                              )                                                                                ⁢                                                      t                            1                                                                                                                                                                          ⋮                                                                                                                          ∑                                                  xe2x80x83                                                ⁢                                                                                                            f                              j                                                        ⁡                                                          (                              t                              )                                                                                ⁢                                                      t                                                          Nd                              -                              1                                                                                                                                                                                                                              ∑                                                  xe2x80x83                                                ⁢                                                                                                            f                              j                                                        ⁡                                                          (                              t                              )                                                                                ⁢                                                      t                            Nd                                                                                                                                              ⁢                                  xe2x80x83                                )                                                                        (        5        )            
The polynomial unit 74 solves expression (5) with respect to each of code vectors C0, C1, . . . , CJ, thereby finding set A defining Nd-order polynomial expression fj(t) for approximation of transitions of distances between learning integrated parameters and code vectors C0, C1, . . . , CJ. Set A is expressed by the following expression:                     A        =                  (                                                                      a                                      0                    ⁢                                          xe2x80x83                                        ⁢                    0                                                                                                a                                      0                    ⁢                                          xe2x80x83                                        ⁢                    1                                                                                                a                                      0                    ⁢                                          xe2x80x83                                        ⁢                    2                                                                              ⋯                                                              a                                                            0                      ⁢                                              xe2x80x83                                            ⁢                      N                      ⁢                                              xe2x80x83                                            ⁢                      d                                        -                    1                                                                                                a                                      0                    ⁢                                          xe2x80x83                                        ⁢                    N                    ⁢                                          xe2x80x83                                        ⁢                    d                                                                                                                        a                                      1                    ⁢                                          xe2x80x83                                        ⁢                    0                                                                                                a                                      1                    ⁢                                          xe2x80x83                                        ⁢                    1                                                                                                a                                      1                    ⁢                                          xe2x80x83                                        ⁢                    2                                                                              ⋯                                                              a                                                            1                      ⁢                                              xe2x80x83                                            ⁢                      N                      ⁢                                              xe2x80x83                                            ⁢                      d                                        -                    1                                                                                                a                                      1                    ⁢                                          xe2x80x83                                        ⁢                    N                    ⁢                                          xe2x80x83                                        ⁢                    d                                                                                                      ⋮                                            ⋮                                            ⋮                                            ⋰                                            ⋮                                                              xe2x80x83                                                                                                      a                                      J                    ⁢                                          xe2x80x83                                        ⁢                    0                                                                                                a                                      J                    ⁢                                          xe2x80x83                                        ⁢                    1                                                                                                a                                      J                    ⁢                                          xe2x80x83                                        ⁢                    2                                                                              ⋯                                                              a                                                            J                      ⁢                                              xe2x80x83                                            ⁢                      N                      ⁢                                              xe2x80x83                                            ⁢                      d                                        -                    1                                                                                                a                                      J                    ⁢                                          xe2x80x83                                        ⁢                    N                    ⁢                                          xe2x80x83                                        ⁢                    d                                                                                )                                    (        6        )            
In step S44, the polynomial unit 74 finds accumulated amounts with respect to each code vector Cj. the values obtained by accumulating an error amount in squares of distance transitions approximated by set A of coefficients in expression (A) and the actual distance transitions. The polynomial unit 74 stores the accumulated amounts, together with set A of coefficients, and proceeds to step S45.
In step S45, the polynomial unit 74 determines whether the accumulated amounts of square errors have been found with respect to all the range of order Nd. If the polynomial unit 74 has determined that the accumulated amounts of square errors have not been found with respect to all the range of order Nd, it proceeds to step S46, and changes the order Nd of polynomial expression fj(t) (expression (6)) to a value on which accumulated amounts of square errors have not been found yet. After that, the polynomial unit 74 returns to step S43, and repeatedly performs the same processing with respect to the changed value of order Nd.
If the polynomial unit 74 has determined in step S45 that the accumulated amounts of square errors have been found with respect to all the range of order Nd, it proceeds to step S47.
In step S47, after detecting a minimum value from among the accumulated amounts of square errors of distance transitions with respect to all the range of order Nd, the polynomial unit 74 outputs the minimum value, together with the stored set A of coefficients defining polynomial expression fj(t), and terminates the processing.
The above-described processing is performed for each word to be recognized.
As described above, by performing time-domain normalization on (time-series) learning integrated parameters, calculating each distance between each time-domain-normalized learning integrated parameter (normal series) and each code vector, and finding a distance-transition model indicating transitions of the distances, a model in accordance with the number of original states of an object to be recognized and state transitions can be obtained. Therefore, according to the obtained distance-transition model, the stationary state and transient state of an object to be recognized can be accurately expressed. As a result, the recognition factor can be increased.
In addition, distance transition is approximated using a polynomial expression. Thus, by using only coefficients defining the polynomial expression, that is, a small amount of data, distance transition can be expressed.
The order Nd of the polynomial expression for approximation of distance transition is determined so that an accumulated value of square errors is minimized. Thus, by using the polynomial expression, the distance transition can be precisely expressed.
The integrated parameters are used as data for learning by the learning apparatus shown in FIG. 9. However, when a single feature parameter is used to perform speech recognition, the single feature parameter may be used for learning.
Because the above-described method of distance-transition model learning may be applied regardless of the type and order of feature parameters, it is not dependent on the type and order of the feature parameters.
In the above-described example, actual distance transition as shown in FIG. 11 is performed such that a distance-transition model (as shown in FIG. 7) approximated by the polynomial expression is used to perform matching. However, matching can be performed, simply using the actual distance transition (as shown in FIG. 11).
With reference to FIG. 13, the spatial distribution matching unit 32 (shown in FIG. 1) is described below.
Integrated parameters are in time series supplied from the integrated parameter generating unit 2 to a vector quantizer 81. The vector quantizer 81 performs vector quantization on the supplied parameters, using a codebook stored in a codebook storage unit, and sequentially supplies a code series as a result of the vector-quantization to a chi-square ("khgr"2) test unit 83.
The codebook storage unit 82 stores the codebook, which is used for vector quantization by the vector quantizer 81. It is not required that the codebook stored in the codebook storage unit 82 (shown in FIG. 13) be identical to that stored in the codebook storage unit 63 of the distance-transition matching unit 63 (shown in FIG. 5).
By performing the chi-square test, the chi-square test unit 83 finds, accessing an expectancy-degree storage unit 84, a test result on whether the spatial distribution of the code series from the vector quantizer 81 is similar to the spatial distribution of a code series obtained when a word to be recognized is pronounced, that is, on whether the integrated parameters supplied to the vector quantizer 81 appropriately correspond to the word to be recognized. The obtained test result is supplied to a sorter 85.
The expectancy-degree storage unit 84 stores degrees of expectancy observed (output from the vector quantizer 81) when the integrated parameters corresponding to the word to be recognized are input (supplied to the vector quantizer 81) with respect to codes corresponding to the code vectors of the codebook stored in the codebook storage unit 82.
The sorter 85 selects upper Nb values in increasing order from among the values of the test result supplied from the chi-square test unit 83, and outputs the selected values as a result of matching by the spatial distribution method to the determining unit 4 (shown in FIG. 1).
The spatial distribution matching unit 32 having the above-described structure performs matching by the spatial distribution method. The process of the matching by the spatial distribution method is described below with reference to the flowchart shown in FIG. 14.
Integrated parameters to be recognized are in time series supplied to the vector quantizer 81. In step S51, the vector quantizer 81 sequentially performs vector quantization on the supplied parameters, and supplies the chi-square test unit 83 with a code series as a result of the vector-quantization.
In step S52, the chi-square test unit 83 finds distribution of the code series supplied from the vector quantizer 81. In other words, the chi-square test unit 83 computes the number of times each code is observed (hereinafter referred to also as the xe2x80x9cobservation degreexe2x80x9d) in the code series supplied from the vector quantizer 81.
In step S53, the chi-square test unit 83 performs the chi-square test, using the observation degree of each code and each expectancy degree (stored in the expectancy-degree storage unit 84) representing an expected number of times each code is observed when a word to be recognized is pronounced, thereby finding each degree (hereinafter referred to also as xe2x80x9csimilarity degreexe2x80x9d) of how much the observation degree of each code on an integrated parameter is similar to the expectancy degree of each code on the word to be recognized. The chi-square test unit 83 outputs the similarity degree of each code to the sorter 85.
Specifically, the chi-square test unit 83 computes chi-square "khgr"2 expressed as the following expression:                               χ          2                =                              ∑                          i              =              0                        J                    ⁢                      xe2x80x83                    ⁢                                                    (                                                      F                    i                                    -                                      f                    i                                                  )                            2                                      f              i                                                          (        7        )            
where Fj represents the observation degree of code #j and fj represents the expectancy degree of code #j in word W. The chi-square test unit 83 supplies, for example, the reciprocal of chi-square "khgr"2 as the similarity degree for word W to the sorter 85.
Chi-square "khgr"2 in expression (7) decreases as the similarity of the observation degree of each code on the input integrated parameter to the expectancy degree of each code on the word to be recognized increases. Therefore, the reciprocal of chi-square "khgr"2, i.e., the similarity degree increases as the similarity of the observation degree of each code on the input integrated parameter to the expectancy degree of each code on the word to be recognized increases.
When receiving, from the chi-square test unit 83, the similarity degrees of all words to be recognized, the sorter 85 selects upper Nb values in order of similarity from among similarity degrees in step S54. In step S55, the sorter 85 outputs the selected values as a result of matching by the spatial distribution method to the determining unit 4, and terminates the processing.
According to the above-described matching by the spatial distribution method, the use of the chi-square test finds similarity between the distribution state of input integrated parameters and the distribution state of integrated parameters on a word to be recognized in the integrated parameter space. Thus, the similarity is not influenced by a time-related change in the input speech. In other words, although the duration of the input speech (the length of a speech period) influences the observation degree used in the chi-square test, a result of the chi-square test is not influenced because it is expected that the observation degree on each code increases or decreases by a value proportional to the length of the speech period. Thus, speech recognition can be performed irrespective of time components in the speech. As a result, the recognition factor can be increased.
FIG. 15 shows an example of a learning apparatus for learning that finds the expectancy degree (stored in the expectancy-degree storage unit 84 shown in FIG. 13) of each code on each word to be recognized.
A vector quantizer 91 is supplied with (time-series) learning integrated parameters on a word to be recognized. The vector quantizer 91 performs vector-quantization on the supplied parameters, using a codebook stored in a codebook storage unit 92, and supplies a code series as a result of the vector-quantization to an expectancy-degree computing unit 93.
The codebook storage unit 92 stores a codebook identical to that stored in the codebook storage unit 82 (shown in FIG. 13).
The expectancy-degree computing unit 93 computes the number of times each code is observed in the code series supplied from the vector quantizer 91.
With reference to the flowchart shown in FIG. 16, a process by the learning apparatus (shown in FIG. 15) is described below.
For one word to be recognized, for example, a plurality of (time-series) learning integrated parameters (obtained such that different speakers pronounce the word, or the same speaker pronounces the word a plural number of times) are supplied to the vector quantizer 91. In step S61, variable i counting the number of the integrated parameters is initialized to, for example, 1, and the process proceeds to step S62.
In step S62, among the learning integrated parameters, the first learning integrated parameter is supplied to the vector quantizer 91, and the vector quantizer 91 performs vector-quantization on the supplied parameter. A code series as a result of the vector-quantization is supplied to the expectancy-degree computing unit 93, and the process proceeds to step S63.
In step S63, the number of times each is observed in the code series supplied from the vector quantizer 91 is accumulated, whereby the observation degree of each code is found.
In step S64, the process determines whether the next learning integrated parameter to be processed is detected. If the process has determined that the next learning integrated parameter to be processed is detected, it proceeds to step S62, and variable i is incremented by 1. The process proceeds to step S62, and the subsequent learning integrated parameter is similarly processed. Thereby, the observation degree of each code corresponding to a plurality of learning integrated parameters is accumulated.
If the process has determined that the next learning integrated parameter to be processed is not detected, it proceeds to step S66.
In step S66, and the accumulated value, obtained by accumulating the observation degree of the each code, is divided by variable i, in other words, a mean of the observation degree of each code from the learning integrated parameters on a word is found. The mean is output as the expectancy degree of each code on the word, and the process ends.
The process shown in FIG. 16 is performed for each word to be processed.
According to the speech recognition apparatus shown in FIG. 1, normalization coefficients for performing normalization so that feature parameters of different media such as image and speech can be treated having the same weight are found beforehand, and feature parameters normalized using the normalization coefficients are used to generate integrated parameters. However, when speech recognition is performed, the use of the integrated parameters may not always be proper.
In other words, feature parameters are normalized using fixed normalization coefficients, and further integrated by the speech recognition apparatus shown in FIG. 1. Thus, even if image feature parameters and speech feature parameters have noticeable characteristics (denoted by arrows) at a time, respectively, as shown in FIG. 17, graph (A) and graph (B), integration using fixed normalization coefficients may cause a situation in which both the image element and the speech element in the integrated parameters cancel each other (as shown in FIG. 17, graph (C)) and their noticeable characteristics are lost. In FIG. 17, in each graph, the horizontal axis indicates time, and the vertical axis indicates the reciprocal of a distance between a code and a normal pattern (e.g., a centroid vector, etc., of the codebook stored in the codebook storage unit 63 shown in FIG. 5) minimizing the distance to a feature parameter at each time.
Accordingly, in order to increase the recognition factor, it is preferable to generate integrated parameters by integrating the feature parameters (in optimal form) so that their characteristics (denoted by arrows in FIG. 17, graph (D)) cannot be lost, as shown in FIG. 17, graph (D) showing integrated parameters caused by time-varying normalization coefficients and integrated signal generation using superior characteristics of media. For generating the integrated parameters, it is required to use, at each time, (time-series) normalization coefficients by which integrated parameters reflect a plurality of feature parameters.
Nevertheless, when speech recognition is performed using integrated parameters obtained by simply arranging (combining) audio feature parameters and image feature parameters, the recognition is greatly affected by either the speech or the image. In other words, the feature parameters of either type have a larger weight, which may prevent the recognition factor from increasing.
Accordingly, it is an object of the present invention to provide an information processing apparatus that increases recognition performance by treating, in optimal form, feature parameters based on different types of inputs such as speech and an image.
To this end, according to an aspect of the present invention, the foregoing object is achieved through provision of an information processing apparatus including a feature parameter detector for detecting feature parameters based on a plurality of input data, a normalizer for normalizing the feature parameters detected by the feature parameter detector while maintaining their feature components, and an integration unit for integrating the feature parameters normalized by the normalizer.
According to another aspect of the present invention, the foregoing object is achieved through provision of a learning apparatus including a normalizer for normalizing, based on first normalization information preset for a plurality of time-series input data, feature parameters of the input data, a detector for detecting a distance between a normal parameter and each of the normalized feature parameters, and a normalization information generator for generating, based on each detected distance, second time-series normalization information for each of the feature parameters.
According to a further aspect of the present invention, the foregoing object is achieved through provision of a learning apparatus including a feature parameter detector for detecting feature parameters based on a plurality of input data, a first normalizer for normalizing the feature parameters detected by the feature parameter detector among the feature parameters, a second normalizer for normalizing the feature parameters normalized by the first normalizer based on the order thereof, a matching unit for performing a matching process on each of the feature parameters normalized by the second normalizer, and a normalization-information generator for generating, based on the result of each matching process from the matching unit, time-varying normalization information.
According to a still further aspect of the present invention, the foregoing object is achieved through provision of an information processing method including the steps of detecting feature parameters based on a plurality of input data, normalizing the feature parameters while maintaining their feature components, and integrating the normalized feature parameters.
According to yet another aspect of the present invention, the foregoing object is achieved through provision of a learning method including the steps of normalizing, based on first normalization information preset for a plurality of time-series input data, feature parameters of the input data, detecting a distance between a normal parameter and each of the normalized feature parameters, and generating, based on each detected distance, second time-series normalization information for each of the feature parameters.
According to a further aspect of the present invention, the foregoing object is achieved through provision of a learning method including the steps of detecting feature parameters based on a plurality of input data, normalizing the detected feature parameters among the feature parameters, further normalizing the normalized feature parameters normalized based on the order thereof, performing a matching process on each of the further normalized feature parameters, and generating, based on the result of each matching process, time-varying normalization information.